This Page Is Inserted by IFW Operations 
and is not a part of the Official Record 



BEST AVAILABLE IMAGES 



Defective images within this document are accurate representations of 
the original documents submitted by the applicant. 

Defects in the images may include (but are not hmited to): 



BLACK BORDERS 

TEXT CUT OFF AT TOP, BOTTOM OR SIDES 
FADED TEXT 
ILLEGffiLE TEXT 
SKEWED/SLANTED IMAGES 
COLORED PHOTOS 

BLACK OR VERY BLACK AND WHITE DARK PHOTOS 
GRAY SCALE DOCUMENTS 



IMAGES ARE BEST AVAILABLE COPY. 



As rescanning documents will not correct images, 
please do not report the images to the 
Image Problem Mailbox. 



5///: cie/n% m f/cp/^c 1 7 

A METHOD AND SYSTEM FOR ORGANIZING DOCUMENTS 
BASED UPON ANNOTATIONS IN CONTEXT 

BACKGROUND OF THE INVENTION 

1 . Field of Invention 
This invention is directed to a document organizing 

system. In particular, this invention is directed to a 
method and a system for organizing documents based upon the 
context of annotations made to those documents. 

2 . Description of Related Art 
When people read paper documents, they often make 

annotations to highlight interesting or controversial 
passages and to record their reactions. Common annotations 
include margin notes, vertical bars, stars, circles, 
underlines, highlights, etc. Two advantages of annotating 
directly on the page are its low overhead and convenience. 
One disadvantage is that the recorded information is hidden 
and inaccessible until the reader returns to the specific 
page in the specific document. 

To avoid this problem, some readers use a separate 
reading notebook to record their annotations. A reading 
notebook is useful because it provides a separate summary 
of what the user has read along with any commentary. The 
advantage of a reading notebook is that it permits a quick 
review of the material because it generally has less 
information to browse and search than the original 
document. One disadvantage of a reading notebook, however, 
is that the reader must recreate the context for each note 
to fully understand the meaning of each note. 

Readers also use note cards to organize notes. The 
advantage of a note card system is that the cards can be 
easily reorganized. However, as with a reading notebook, 
unless the reader recreates it, there is no context 
available to permit the user to fully understand the notes. 
Additionally, each note must be categorized onto the 
correct card before it can be recorded. 
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Handwritten notes and keywords are used in a system 
known as "Marquee" to index video. This system is 
described in "Marquee: A Tool for Real-Time Video Logging", 
K. Webber et al.. Proceedings of CHI '94 . April 1994, pp. 
58-64, incorporated herein by reference in its entirety. 
In "Marquee", notes are synchronized to a video string with 
time zones that are created with horizontal line gestures. 
Keywords are identified by the user by circling the words 
and notes that the user has selected as keywords. The 
keywords are assigned to the time zone in which the keyword 
is created. Keywords also may be assigned directly by the 
user by typing the keyword in manually. Because the 
keywords are associated with time the user can view an 
index of time zones and go directly to the video by 
15 selecting a time zone using an index" of the previously 
identified keyword or annotations. Although "Marquee" uses 
annotations to index a video document, it does not combine 
the annotations with the document in a visual way. 
"Marquee" is thus analogous to notetaking in a separate 
20 notebook rather than on the document itself. 

"Dynomite" is a free-form digital "ink" notebook. 
The digital ink notebook is a pen-based computer that the 
user controls by writing with a pen directly on the screen 
of the computer. The computer senses the location and the 
25 positions traversed as the pen moves across the display and 
assigns ink marks that correspond with the positions of the 
pen. These ink marks are called digital ink because the 
ink is described by the computer digitally. Dynomite 
extracts the ink, assigns properties to each ink mark and 
30 can present a list of the ink marks sorted by the assigned 
properties. This list is known as an ink index. This 
system is described in co-assigned and co-pending United 
States patent application. Serial No. 08/821,311, filed on 
March 20, 1997, entitled "System for Capturing and 
35 Retrieving Audio Data and Corresponding Handwritten Notes", 
and "Dynomite: A Dynamically Organized Ink and Audio 



Notebook", by L. Wilcox et al . , m CHI '97 Conference 
Proceedings , ACM Press, 1997, pp. 186-193, 1^:^^^;^;:^ 
herein by reference in their entireties. This ink index 
shows a "type" of the "ink" along with a time stamp and 
provides links to the original notebook pages. Dynomite's 
xnk index provides "ink" marks linked to the corresponding 
full notebook page. However, Dynomite organizes only the 
ink notes themselves and not the associated information. 

"ComMentor" is a platform for shared annotations 
that attaches text-based comments to locations within web 
documents. This system is described in "Shared Web 
Annotations as a Platform for Third-Party Value-Added 
Information Providers: Architecture, Protocols, and Usage 
Examples", by M. Roscheisen, et al.. Technical Report STAN- 
CS-TR-97-1582, Stanford Integrated Digital Library Project, 
Computer Science Department, Stanfotd University, November 
1994, Updated April 1995, incorporated herein by reference 
in its entirety. Annotations are grouped into sets. A 
user can filter these sets and tour through documents 
within a set. A tour window shows a list of annotations, 
each annotation shown with the document title of the 
annotated document and a number of annotation attributes. 
Clicking on the annotation causes the display to jump to 
the source document at the position of the annotation. 
ComMentor uses filtered annotations to produce lists of 
read documents, but does not support paper-like annotations 
or present lists of annotations in context. 

Classroom 2000 is a system for capturing a lecture 
using recorded audio, prepared visual materials and 
handwritten notes made on a display overlay of viewgraphs. 
This system is described in "Classroom 2000: Enhancing 
Classroom Interaction and Review", by G. Abowd et al., In 
Proceedings of CSCW '96, March 1996, incorporated herein by 
reference in its entirety. Searching the text in the 
viewgraphs retrieves the viewgraphs along with the overlaid 
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The Freestyle system, which was developed at Wang 
Laboratories, is a mechanism for sketching and writing on 
screen snapshots or on sheets of electronic paper. 
Freestyle records cursor movement and audio as well as the 
handwriting. This system is described in "Rapid Integrated 
Design of a Multimedia Communication System, and Human- 
Computer Interface Design", E. Francik, Marianne Rudisill 
et al. (editor), Morgan Kaufman Publishers, Inc., 1996, 
incorporated herein by reference in its entirety. The 
result is a dynamic multimedia message that can be mailed 
to others. Freestyle does not provide the ability to 
organize the handwritten annotations. 

The PENPOINT operating system for pen-based 
computers, recognizes pen gestures for editirvg and allows 
arbitrary "ink" marks to be placed on top of any document 
using an "acetate layer". This system is described in "The 
Power of PENPOINT", by R. Carr et al., Addison-Wesley, 
Inc., 1991, incorporated herein by reference in its 
entirety. Although both Freestyle and PENPOINT support 
free- form document annotation, neither provides any way to 
retrieve documents based upon those annotations. 

In 1945 Vannevar Bush described a vision of a mesh 
of trails running through a mechanized private file and 
library or memex in "As We May Think", in Atlantic Monthly , 
July 1945, pp. 101-108, incorporated herein by reference in 
its entirety. These trails were produced as part of the 
reading activity, and provided a way to create and share 
personal organizations of information. Bush's visions were 
seminal in the development of hypermedia systems such as 
Engelbart's NLS and the World Wide Web. However, 
hypermedia systems have focused on sharing, browsing and 
more explicit authoring of links, not on personal 
organization and annotation. 

Thus, an annotation system for electronic documents 
is needed that combines the advantages of marking directly 
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on a document with quick accessibility and the flexible 
organization of marking on note cards or in a notebook. 

SUMMARY OF THE INVENTION 
This invention provides a system and method for 
using digital "ink" for annotations in context to organize 
a reader's activities. The system and method of this 
invention extracts the contents surrounding and underlying 
a reader's annotations and presents this information to the 
reader with links to the full context. The annotations in 
context provided by the system and method of this invention 
permits flexible low-overhead organization of material 
without adding to the effort of reading and notetaking. 

These and other features and advantages of this 
invention are described in or are apparent from the 
15 following detailed description of the preferred 
embodiments. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The preferred embodiments of this invention will be 
described in detail, with reference to the following 
figures, wherein: 

Fig. 1 is a block diagram of the doctiment 
organizing system of this inventions- 
Fig. 2 is a flow chart outlining the control 
routine of one embodiment of this inventions- 
Fig. 3 shows a document annotated according to this 
inventions- 
Fig. 4 shows the annotated portions of the document 
of Fig. 3 

Fig. 5 shows another view of the annotated document 
30 of Fig. 3; and 

Fig. 6 is a flow chart outlining the annotation 
control routine of one embodiment of this invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
Fig. 1 is a block diagram of one embodiment of the 
electronic document organizing system 10 of this invention. 
The system 10 has a processor 12 communicating with a 
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display 14, a first storage device 16, a second storage 
device 18 and an input/output interface 20. The first 
storage device 16 stores a document 22 displayable on the 
display 14. The input/output interface 20 communicates 
with any number of conventional input/output devices 24 
such as a mouse 26, a keyboard 28 and/or a pen-based 
device 30. A user manipulates the input/output devices 24 
to annotate the document 22 when displayed on the 
display 14. The system 10 then stores these annotations 32 
in the second storage device 18. 

As shown in Fig. 1, the system 10 is preferably 
implemented using a programmed general purpose computer. 
However, the system 10 can also be implemented using a 
special purpose computer, a programmed microprocessor or 
microcontroller and any necessary peripheral integrated 
circuit elements, an ASIC or other integrated circuit, a 
hardwired electronic or logic circuit such as a discrete 
element circuit, a programmable logic device such as a PLD 
PLA, FPGA or PAL, or the like. In general, any device on 
which a finite state machine capable of implementing the 
flowchart shown in Fig. 2 can be used to implement the 
system 10. 

Additionally, as shown in Fig. 1, the memories 16 
and 18 are preferably implemented using static or dynamic 
RAM. However, the memories 16 and 18 can also be 
implemented using a floppy disk and disk drive, a writable 
optical disk and disk drive, a hard drive, flash memory or 
the like. Additionally, it should be appreciated that the 
memories 16 and 18 can be either distinct portions of a 
single memory or physically distinct memories. 

Furthermore, it should be appreciated that the link 
17 connecting the memory 16 and the processor 10 can be a 
wired or wireless link to a network (not shown). The 
network can be a local area network, a wide area network, 
an intranet, the internet or any other distributed 
processing and storage network. m this case, the 
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electronic document 22 is pulled from a physically remote 
memory 16 through the link 17 for processing in the 
processor 10 according to the method outlined below. m 
this case, the electronic document 22 can be stored locally 
in a portion of the memory 18 or some other memory (not 
shovm) of the system 10. 

The method of this invention includes three 
distinct processes. First, the reader makes annotations on 
a displayed document, and the annotations are extracted 
along with their context. Second, the system associates a 
number of attributes with the annotations in order to 
facilitate retrieval of the annotations and/or the 
underlying annotated documents. Third, the reader views 
collections of the annotations in context, where the 
15 collections are organized by those attributes. 

The system 10 records annotations on electronic 
documents. A preferred interface for entering the 
annotations is a pen-based computer, where the reader 
"writes" directly on the electronic document. On a desktop 
computer without a pen, clicking a mouse in a margin might 
create a text overlay box to create the annotation. The 
system 10 may also support a number of different styles of 
marking. For example, these styles can include swiping 
with a highlighter pen, underlining text, vertical bars in 
25 the margin, circled regions, and margin notes. 

Fig. 2 is a flow chart outlining a control routine 
of one embodiment of the invention. The control routine 
starts at step SlOO and proceeds to step SllO, where the 
user marks on the display of the document with digital ink 
30 to annotate it. The control routine then proceeds to 
step S120, where the system groups the marks of the digital 
ink by time and/or space into collections of marks, treated 
as a single annotation as will be described in more detail 
below. Next, the control routine proceeds to step S130, 
where the system determines the minimum context for each 
annotation. The system has a minimum context that 
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determines how much of the document that surrounds the 
annotation is to be associated with the annotation. The 
minimum context may be predetermined as a user preference 
to be a few words, a sentence, a paragraph or any other 
amount in accordance with the user's preferences. The 
minimum context can be displayed to the user as a bounding 
box around the minimum context. The bounding box encloses 
the bounding region and the minimum context is defined as 
the content enclosed within the bounding region of the 
corresponding annotation. Segmentation procedures are 
applied to the document to divide it into graphical 
components, e.g., lines of text, sentences, paragraphs and 
figures. . Given the minimum context, the control routine 
expands the context to include all of the nearby segments. 
With this procedure, the context may include a couple of 
lines, the surrounding sentence, or the entire surrounding 
paragraph. Fig. 5 shows a bounding box 34 with the context 
around a circle annotation 33. 

The annotation control routine is shown in Fig. 6. 
The control routine starts at step S200 and proceeds to 
step S210 where the user selects and opens an electronic 
document. The user then starts marking on the document at 
step S220 and creates digital ink. The system then 
determines at step S230 if the new ink is close enough in 
time and space to be associated with previous ink marks. 
The system has time and space thresholds that may be 
predetermined or adjusted in accordance with a user's 
preferences. If the system determines at step S230 that 
the ink marks are not separate the system proceeds to step 
S240 where the user continues to mark. As each mark is 
entered by the user steps S230 and S240 are repeated until 
the system determines that the new ink is separated enough 
by time and space to proceed to step S250. At step S250 
the ink marks are grouped together as a single annotation 
and at step S260 the context for the annotation is 
determined and the attributes are assigned to the 
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annotation. The control routine then proceeds to step S27 0 
where the system determines if a new mark has been input. 
If a new mark has been input the control routine returns to 
step S230. If no new mark is entered at step S280 then the 
annotations are organized and displayed. The control 
routine then stops at step S290. 

For some special annotation formats such as those 
shown in Fig. 5, the control routine determines the context 
slightly differently. For margin bars 36 and other notes 
in the margin 38, the system ignores the horizontal 
distance when finding nearby segments. Thus, all 

vertically adjacent material is included in the contexts 40 
and 42, respectively. For the line callouts and circle 
callouts, the control routine determines the minimum 
15 contexts and from the underlined or circled text, etc., 
ignoring the ink in the callout gesture. 

After the context of each annotation has been 
determined, the control routine proceeds to step S140, 
where the control routine assigns attributes to the 
annotations in at least one of three ways: 1) attributes 
entered by the user; 2) attributes inherited from the 
document's attributes; and 3) implicit or explicit 
attributes derived from the annotations themselves. 

The user may enter attributes by interacting with a 
dialog box or by selecting from a marking menu, or by 
selecting a special pen. Example, attributes derived from 
the annotations themselves include "agree", "disagree", 
"good idea", and "follow-up". m addition, annotation 
gestures such as "exclamation point" and "question mark" 
may be interpreted to mean "good idea" and "questionable" 
by the system as they are entered on the page. Attributes 
may also be entered implicitly, the most important of which 
is the date and time that the annotation was made and the 
page number at the annotation. Another implicit attribute 
35 is the form of the annotation, e.g., highlight, circle, 
marginal note, etc. 
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Attributes may also be inferred from documents. In 
the system 10, the electronic documents are already 
associated with a variety of attributes, such as creation 
date, author, providence and title. 

After the attributes are assigned to each 
annotation at step S140, the control routine proceeds to 
step S150, where the annotations are organized, ordered or 
ranked by the assigned attributes. Subsequently, the 
control routine proceeds to step SI 60, where the 
annotations are displayed for the user. The control 
routine then proceeds to step S170, where the control 
routine stops. 

The system 10 visually presents the annotations in 
context using different list views. Lists are ordered or 
15 filtered by the attributes described above. The system 10 
allows the reader to navigate between these views and the 
underlying electronic documents. Examples of ordered lists 
include: 

1) Ordered by time. This view is analogous to 
20 a reader's notebook, but also automatically includes the 

context of each annotation, as shown in Fig. 4, without 
further effort by the user. 

2) Filtered by attributes. Passages across a 
number of documents are listed in one view; 

3) Filtered by the type of adjacent material. 
For example, annotations of pictures along with the 
pictures themselves; and 

4) Filtered by the content of adjacent 
material. For example, annotated passages mentioning 
patent leather shoes are ranked in relatedness using 
known information retrieval techniques. 

It is to be understood that the term annotation 
as used herein is intended to include text, digital ink, 
audio, video or any other input associated with a 
35 document. It is also to be understood that the term 
document is intended to include text, video, audio and 
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any other media and any combination of media. Further, 
it is to be understood that the term text is intended to 
include text, digital ink, audio, video or any other 
content of a document to include the document's 
structure . 

While this invention has been described with the 
specific embodiments outlined above, many alternatives, 
modifications and variations are apparent to those 
skilled in the art. Accordingly, the preferred 

embodiments described above are illustrative and not 
limiting. Various changes may be made without departing 
from the spirit and scope of the invention as defined in 
the following claims. 



12 



10 



15 



WHAT IS CLAIMED IS : 

1. A method for displaying context embedded 
annotations from at least one document, comprising: 

extracting at least one annotation from the 
at least one document; 

extracting a context portion for each at 
least one annotation from a corresponding one of the at 
least one document; and 

assigning at least one attribute to each at 
least one extracted annotation. 

2. The method of claim 1, further comprising 
ordering the at least one annotation based on the at 
least one attribute; and 

displaying an ordered list of the at least 
one annotation along with the corresponding context. 

3. The method of claim 1, wherein the step of 
extracting at least one annotation comprises: 

grouping marks by time and space into at 
least one collection, wherein each collection forms an 
20 annotation; 

segmenting the at least one document into a 
plurality of segments based on the at least one 
annotation; 

determining a minimum context portion for 
25 each annotation, from the segments; and 

determining the context portion based on the 
segments surrounding the minimum context. 

4. The method of claim 3, wherein, when one of 
the at least one annotation is a margin note, the step of 

30 extracting the at least one annotation comprises 
determining the context portion by including all segments 
vertically adjacent to the margin note. 

5. The method of claim 3, wherein, when one of 
the at least one annotation is an underline callout, the 

35 step of extracting the at least one annotation comprises 
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determining the minimum context portion based on an 
underlined portion of the corresponding document. 

6. The method of claim 3, wherein, when the at 
least one annotation is a circle callout, the step of 
extracting the at least one annotation comprises 
determining the minimum context from a circled portion of 
the corresponding document. 

7. The method of claim 2, wherein the step of 
assigning at least one attribute comprises assigning at 
least one user-defined attribute to each at least one 
annotation. 

8. The method of claim 2, wherein the step of 
assigning at least one attribute comprises assigning at 
least one document-based attribute to each at least one 

15 annotation. 

9. The method of claim 2, wherein the step of 
assigning at least one attribute comprises deriving at 
least one attribute for each at least one annotation from 
the context portion for that annotation. 

The method of claim 9, wherein the step of 
deriving the at least one attribute comprises identifying 
a type of the context portion for that annotation and 
assigning at least one attribute based on the identified 
type. 

The method of claim 9, wherein the step of 
deriving the at least one an attribute comprises 
identifying a content of the context portion for that 
annotation and assigning at least one attribute based on 
the identified content. 

^° 12. The method of claim 1, wherein the step of 

assigning at least one attribute comprises identifying a 
mark on the document by the user and assigning at least 
one attribute corresponding to the identified mark. 

13. The method of claim 1, wherein the step of 

35 assigning at least one attribute comprises determining a 
date and/or a time of each of at least one annotation and 



assigning the corresponding creation date and/or time as 
an attribute of the annotation. 

14. The method of claim 1, wherein the step of 
assigning at least one attribute comprises determining a 
page number of each at least one annotation and assigning 
the corresponding page number as an attribute of the 
annotation. 

15. The method of claim 1, wherein the step of 
assigning at least one attribute comprises: 

identifying for an annotation, a 
user-generated mark on the document associated with that 
annotation; and 

assigning at least one attribute to that 
annotation based on the . identified mark. 

16. The method of claim 1, further comprising 
filtering the annotations based on the assigned 
attribute. 

17. An apparatus for displaying annotations from 
at least one document, the apparatus comprising: 

a memory that stores the at least one 

document; 

a processor that extracts at least one 
annotation and a context portion of the at least one 
document corresponding to each at least one annotation, 
that assigns at least one attribute to each at least one 
extracted annotation that orders the at least one 
annotations based on the at least one assigned attribute; 
and 

a display that displays an ordered list of 
the at least one annotations and the at least one 
corresponding context portion. 

18. The apparatus of claim 17, wherein: 

the processor extracts the at least one 
annotation grouping at least one user generated mark on 
the at least one document based on time of at least one 
mark and a location within the document of the at least 
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one mark into at least one collection, wherein each 
collection forms a single annotation, the processor 
determines a minimum context portion of the corresponding 
document for each annotation; and 

determines the context for each annotation 
based on the annotation and the minimum context. 

19. The apparatus of claim 18, wherein when one 
of the at least one annotation comprises a margin note, 
the processor extracts that annotation by including all 
segments vertically adjacent to the margin note in the 
corresponding document. 

20. The apparatus of claim 18, wherein, when one 
of the at least one annotation comprises an underline 
callout, the processor extracts that annotation by 
determining the minimum context from an underlined 
portion of the corresponding document. 

21. The apparatus of claim 18, wherein, when one 
of thie at least one annotation comprises a circle 
callout, the processor extracts that annotation 
determines the context portion based on a circled portion 
of the corresponding docviment. 

22. The apparatus of claim 17, wherein, for each 
at least one annotation, the processor assigns the at 
least one attribute to that annotation based on at least 
one document-based attribute. 

23. The apparatus of claim 17, wherein, for each 
at least one annotation, the processor assigns the at 
least one attribute to that annotation based on at least 
one attribute derived from the context portion 
corresponding to that annotation. 

24. The apparatus of claim 23, wherein, for each 
at least one annotation, the processor derives the at 
least one attribute from the context portion based on a 
type of the context portion corresponding to that 
annotation. 
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25. The apparatus of claim 23, wherein, for each 
at least one annotation, the processor derives the at 
least one attribute from the context portion based on the 
content of the context portion corresponding to that 

5 annotation. 

26. The apparatus of claim 17, wherein when one 
of the at least one annotation is a mark annotation the 
processor assigns the at least one attribute based upon 
that mark annotation. 

27. The apparatus of claim 17, wherein, for each 
at least one annotation, the at least; one attribute is at 
least one of a creation date and a creation time of that 
annotation. 

28. The apparatus of claim 17, wherein, for each 
at least one annotation, the at least one attribute is 
the page ntunber of that annotation. 

29. The apparatus of claim 17, wherein, for each 
at least one annotation, the processor assigns the at 
least one attribute to that annotation based on a user- 
generated mark on the corresponding document associated 
with that annotation. 

30. The apparatus of claim 17, wherein the 
processor filters the annotations using the at least one 
assigned attribute. 
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ABSTRACT OF THE DISCLOSURE 
A document organizing system extracts annotations 
made to a document along with the context surrounding each 
annotation and organizes the annotations based upon the 
5 annotation attributes and/or context. The annotations are 
created by grouping marks based upon their proximity in 
time and space. The document is segmented to determine a 
minimum context associated with each annotation. A list of 
the annotations sorted by the attributes are then displayed 
10 to the user. The context provided by the invention for 
each annotation allows the user to fully understand the 
annotation. 
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